As described in U.S. Pat. Nos. 5,703,961, and 6,327,381, the disclosures of which are hereby incorporated by reference herein, an image of a scene can be captured by cameras at numerous starting viewpoint locations on a viewpoint locus. The image captured at each location contains pixel data. Each item of pixel data represents illumination falling on a particular point within the image. Using the methods disclosed in the aforementioned patents, pixel data from various images can be combined so as to reconstruct a “virtual viewpoint image” representing the image that would have been acquired by a camera at a theoretical or “virtual” viewpoint. For example, the virtual viewpoint may be selected by tracking the movements of a viewer's head or eyes so that the virtual viewpoint and view direction change as the viewer moves. By displaying the virtual viewpoint image to the viewer as the user moves or turns his head, the system provides a close approximation to the view which the user would have if he or she were making the same movements while actually being present within the scene. This effect is commonly referred to as “telepresence.”
Where the scene to be displayed is a static scene, the image that provides the original pixel data can be acquired by one camera or a few cameras which move to the various viewpoints in sequence. However, where the scene is a dynamic scene, a camera typically is provided at each starting viewpoint location so that the array of cameras can acquire new sets of images periodically. For example, each camera can acquire a sequence of images much like a conventional sequence of video images. The images taken at a given time by all of the various cameras constitute a single frame of starting image data. The combining operations discussed above are repeated, desirably in real time, to combine pixel data from various images taken within each frame. Such a camera array can generate vast amounts of data.
Aspects of the present invention provide structures for a camera array, which can be used in a system as discussed above, or for other purposes.
Telepresence systems as discussed above do not require the use of modeling techniques commonly used in virtual reality systems. In modeling techniques, the shape and size of a displayed image of a theoretical object are generated based on data defining the size and shape of the object and the position of the object in space relative to the position of the observer. The brightness of the displayed object is computed from object properties such as reflectivity of the theoretical object at various wavelengths in combination with illumination data defining the light which theoretically falls on the object. Such illumination data may include information about the brightness and color of light and may further include information defining the direction of the light. In a typical virtual reality system, the location and properties of light sources are assumed and the illumination data is derived from this information.
In some instances, it is desirable to superimpose a virtual reality display on a telepresence display. For example, in a police training system, a telepresence system may be employed to capture and show a scene in a real building so that a trainee can move through the building in a realistic manner. It may be desirable to superimpose a virtual reality display of a criminal or contraband on the display of the building. In an entertainment system, it may be desirable to display an image of a creature created by virtual reality on a natural scene displayed by a telepresence system.
A problem which can arise in such combined systems is that virtual reality portion of the display may have an unnatural appearance if the assumed illumination used in the virtual reality system differs from the actual illumination on the real scene displayed in the telepresence portion of the display. For example, if a cloud momentarily blocks the sun, the brightness of a displayed natural scene will decrease. If the assumed illumination used in the virtual reality system remains constant, the objects generated by the virtual reality system will appear to grow brighter relative to the remainder of the scene.
This illumination problem also occurs where one real image is to be superimposed over another. For example, a first real scene may be recorded. It may then be desirable to superimpose a second real scene over the first. The second real scene may be an image captured using blue or green screen methods known in the art. In order for the second real scene to look more natural when superimposed over the first real scene, the blue screened image may be illuminated using data collected from the actual scene. The illumination may also be adjusted after the data is collected using computerized modeling techniques known in the art. One such technique is to take the mathematical difference between the illumination of the first real scene and the apparent illumination of the second real scene and apply a corrective illumination. Thus, the brightness of the superimposed image will match that of the remainder of the scene.
A further aspect of the invention provides methods of capturing a scene for telepresence systems and other video systems. In methods according to this aspect of the invention, information about real illumination impinging on the scene is captured substantially simultaneously with the scene. This illumination information may then be used to adjust the apparent illumination of a second image superimposed over the scene.
A further aspect of the invention provides a data storage medium as, for example, a disc, tape or memory, containing stored image data specifying one or more images of a scene and illumination data representing the illumination impinging on the scene at the time the images were captured. A further aspect of the present invention provides an enhanced camera array which can capture information about ambient lighting along with the information constituting one or more images of the scene. A further aspect of the invention provides for processing of the collected data.